[RFC] Power management #173

pmenzel · 2026-07-19T13:36:54Z

Everything still work in progress.

Separate daemon controlling nodes to power them off and start them.

We currently don't start mxqd autmatically after boot. This would need to be changed.
We would not be able to use the provided .timer as-is because it should be enabled on a single Mariux64 node only. Maybe on afk , maybe somewhere else with a ssh key which allows the required commands do stop mxqd or shutdown only.

…shes mxqsub accepts --callback=EXECUTABLE (absolute path). The daemon runs the executable after every terminal outcome (finished, failed, killed, unknown), as the submitting user in the job workdir, with MXQ_JOB_ID, MXQ_GROUP_ID, MXQ_JOB_STATUS, and MXQ_JOB_WORKDIR in the environment. The callback is double-forked so it does not block the daemon, and runs under RLIMIT_CPU=60s, RLIMIT_AS=256MiB, RLIMIT_CORE=0. Adds job_callback column to mxq_job; migration in migrate_019. Also fixes mxq_set_job_status_unknown not updating job->job_status in-struct, inconsistent with the other status-update functions. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

… demand) Add mxq-powerd, a self-contained power manager for the MXQ cluster that powers idle compute nodes down to save energy and powers them back up when queued work needs them, oldest/least-efficient hardware off first. It runs on the management host on a fixed interval (systemd timer or cron). The manager is entirely external to MXQ: it only READS the MySQL database (through the stock mysql client) and actuates nodes with stock CLIs (ipmitool, wakeonlan, ssh). It issues no DB writes and requires no changes to mxqd, the schema, or mxqadmin. Implements docs/power-management-implementation.md: - static inventory /etc/mxq/nodes.conf (capability + transport + priority) - global policy /etc/mxq/powerd.conf (dry_run defaults to true) - IPMI credentials in /etc/mxq/powerd-secrets (0600; password via ipmitool -E, never on the command line) - per-node state machine in /var/lib/mxq/powerd-state.json, advanced each tick from the DB snapshot (pending demand, live capacity/liveness, inferred idle duration) - resume: wake lowest-gen nodes on core deficit / capability gap / warm pool shortfall, rate-limited - suspend: drain highest-gen idle nodes (ssh mxqdctl-hostconfig stop), wait for EXITED to close the claim race, then poweroff, with BMC soft/hard-off escalation; honours the surplus test, warm pool floor and rate limits - bounded transitions: stuck nodes land in FAILED, are excluded, and are left for an operator (mxq-powerd --clear HOST) Ships example configs, systemd service + timer, an mxq-powerd.8 man page, a README, and an opt-in `make install-powerd` target (not wired into the default install). DB access reuses MXQ's read-only defaults file (/etc/mxq/mysql_ro.cnf, group mxqclient) via my_print_defaults rather than adding a Python MySQL driver dependency; selectable in powerd.conf. Installs the program to /usr/libexec/mxq to match the existing helper convention. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

donald · 2026-07-20T05:18:46Z

powerd/mxq-powerd

+            return False
+        if node.gpus < group.gpu:
+            return False
+        if group.tags and not group.tags.issubset(node.tags):


That's not correct. I think, group tags are evaluated against host prerequistes expressions and visa versa, not tags against tags. Also, is blacklist/whitelist missing?

Same in the following _alive_can_serve()

Sorry, I have limited access currently (no time, on mobile) can't really look into this now.

The one parameter we don't know have to decide whether a node can accept a job is the free tmpdir size. It's PITA that we allow that resource to be consumed by users outside of mxqds control via /scratch/local2 on the same filesystem.

donald · 2026-07-20T05:22:05Z

We currently don't start mxqd autmatically after boot. This would need to be changed.

donald · 2026-07-20T05:35:16Z

We would not be able to use the provided .timer as-is because it should be enabled on a single Mariux64 node only. Maybe on afk , maybe somewhere else with a ssh key which allows the required commands do stop mxqd or shutdown only.

@donald

Reviewer feedback (@donald) on the RFC: the job/host match was wrong. It tested group tags as a subset of node tags, but MXQ (server_is_qualified() in mxqd_control.c) does something quite different, and the host white/blacklist was missing entirely. Reproduce the real model for powered-off nodes: * group_whitelist / group_blacklist gate on the hostname; * the group's `prerequisites` EXPRESSION is evaluated against the host's tags, and the host's own `prerequisites` against the group's tags; * a GPU group needs a GPU host. - Add a small evaluator mirroring parser.y (tags as identifiers, `! & |`, parens, precedence ! > & > |; empty => true, syntax error => false). - Pull tags, prerequisites, group_whitelist and group_blacklist per pending group; tokenise tag/white/blacklist strings on whitespace like keywordset_new(). - Give nodes.conf a `prerequisites` column and treat each node's tags as including the implicit `true`, hostname and short hostname that mxqd adds. - Fold qualification + a capacity check into _host_satisfies()/_alive_can_serve(). Also document the operational points raised in review: enable the timer on a single management node, mxqd must start at boot for resume to work, use an ssh key restricted to the drain/poweroff commands, and the free-tmpdir blind spot. README, man page and design doc updated to match. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

pmenzel · 2026-07-20T08:31:28Z